Fix progress logging with Dask v2026#3154
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3154 +/- ##
=======================================
Coverage 96.25% 96.25%
=======================================
Files 273 273
Lines 16188 16194 +6
=======================================
+ Hits 15581 15587 +6
Misses 607 607 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
schlunma
left a comment
schlunma
left a comment
There was a problem hiding this comment.
Very nice! I tested all cases I could think of (cartesian product of dask.use="local_threaded"/"local_distributed", max_parallel_tasks=1/2, log_progress_interval=-1/0/1). Everything looks as expected!
Just one tiny comment on the code.
|
Regarding the label: This is again an example of a PR that fixes something that was never broken in a release (ESMValGroup/ESMValTool#4389). We still do not have an appropriate label for this. |
bouweandela
added
the
issue introduced since last release
|
Thanks for reviewing ✨ I created the label with the name @ehogan suggested and added it to this pull request. |
|
unreal - I am stumped to see so much code change needed to make a progress bar work - Dask is really giving us more grief than joy. Many thanks @bouweandela and @schlunma - sorry I was tied up elsewhere today 🍺 |
|
It's mostly technical debt on our side, i.e. #2954 and missing type hints. |
bouweandela
changed the title
bouweandela
added
installation
|
Relabelled this as |
3 participants
Description
Update the way the Dask progress loggers work:
esmvalcore.config.Sessioninstead of relying on the globalesmvalcore.config.CFGvariable, which will revert to the default in a different process, when setting up the progress bars. This fixes the issue that whenmax_parallel_tasksis set to a different value from the command line, the right kind of progress bar (Rich based if max parallel tasks = 1 and logger style if larger than 1) is not selected.I had to make quite a few changes in order to pass the session down to where logging progress is set up.
Closes #3121
Helps with #2954
Before you get started
Checklist
It is the responsibility of the author to make sure the pull request is ready to review. The icons indicate whether the item will be subject to the 🛠 Technical or 🧪 Scientific review.
To help with the number pull requests: